Techniques for accurate pose estimation in outdoor environments

ABSTRACT

The described technology regards an augmented reality system and method for estimating a position of a location of interest relative to the position and orientation of a display, using a forward buffer to store current and predicted position estimates calculated by the methods of the present invention. Systems of the described technology include including a plurality of sensors, a processing module or other computation means, and a database. Methods of the described technology use data from the sensor package useful to accurately render graphical user interface information on a display.

The inventions herein disclosed were made with government support underFederal Contract No. FA8650-09-C-7909, awarded by DARPA (DefenseAdvanced Research Projects Agency). The government has certain rights tothe inventions described in this application.

BACKGROUND OF THE TECHNOLOGY

The described technology regards vision-aided navigation, and inparticular pose estimation useful in vision aided navigation,advantageous in wearable augmented-reality (AR) systems operating innatural outdoor environments and other applications.

While a variety of GPS/INS solutions achieve pose estimation, currentlyavailable solutions do not offer the level of customization needed for aperson walking over arbitrary outdoor environments. The systems andmethods of the described technology take advantage of kinematicsconstraints, and detection of magnetic and dynamic disturbances, toachieve enhanced accuracy. Further, the use of digital terrain andelevation data (DTED) information allows systems of the describedtechnology to provide a stable measurement of altitude with measurementsof absolute orientation, systems incorporating the described technologyreach levels of performance that are not possible with other GPS/INSsystems.

GENERAL DESCRIPTION

The described technology regards pose estimation systems useful inaugmented-reality systems and other technology, including a plurality ofsensors, a processing module or other computation means, and a database.

The plurality of sensors or sensor package may include sensors such as acamera, a 3-axis accelerometer, a 3-axis angular rate gyro, a 3-axismagnetometer, a barometric pressure sensor, and a GPS receiver, and maybe mounted to a rigid reference assembly. Data from the sensor package,which could include delayed measurements, is transmitted to theprocessing module or other computation means, which generates signalsthat render graphical user interface information on a display using thesensor data. The processing module also is communicatively coupled with,and uses information from a database, which receives, stores, andtransmits data such as locations of interest in geodetic coordinates(latitude, longitude, altitude), and digital terrain and elevation data.The database may be local to the processing module, virtual, stored on asingle or network of remote servers, on the Internet or otherwiseaccessible to the processing module.

The data received by the processing module may include time-stampedsensor values such as camera imagery, accelerometer measurements,rate-gyro measurements, magnetometer measurements, barometric pressuresensor measurements, GPS receiver position measurements, and GPSreceiver velocity measurements. From this data and data representing thespatial position of a display relative to the spatial position of thesensors, the processing module calculates over time a position vectorrepresenting the current estimated position of a location of interestwith respect to the position and orientation, such as the orientation,geodetic position (longitude, latitude, altitude), or combinationthereof, of the display, expressed in display coordinates. In someembodiments the position vector derives from the sensor values receivedby the processing module, using an Extended Kalman Filter (EKF)structure based on a retroactive update to a previously rendered poseestimate stored in a rewind buffer, or a forward buffer to store currentand predicted pose estimates calculated by the processing module, forexample by means of circuitry, firmware and/or a processor. Each delayedmeasurement may be stored in the rewind buffer and processed using aseparate sequential update. In some embodiments the processing moduledetects the presence of magnetic disturbances and, if detected, rejectsmagnetometer measurements corresponding to magnetic disturbances. Insome embodiments the calculation of the position vector uses an EKFstructure having an adjust-update-predict (AUP) cycle, which adjusts theprevious prediction pose values by including the effects of currentinput data in addition to current measurement data to estimate thecurrent state.

The processing module then generates signals to render on a display, ata position derived from the position vector, graphical user interfaceinformation including a graphical icon representing the location ofinterest. The processing module may implement a baseline GPS/INS tocalculate the position vector, and may adjust the baseline GPS/INS withabsolute orientation information when available.

Some embodiments of the described technology further include a radio tosend and receive geo-spatial data relating to objects in theenvironment. The processing module receives, stores and uses suchgeo-spatial data, and generates signals to render graphical userinterface information on a display such as a graphical icon representingsome of the objects in the environment at a position derived from theposition vector. In some embodiments of the described technology arate-gyro measurement vector received by the processing module isselectively filtered by the processing module based on its magnitude.

The described technology further regards a method for providing anestimate of position for a location of interest relative to a display.The method includes the steps of receiving sensor data from a pluralityof sensors which may include delayed data measurements and datarepresenting the pose of a display, and applying an Extended KalmanFilter (EKF) to the received data and the geodetic coordinates of alocation of interest, to estimate a pose of the display from which aposition of the location of interest relative to the pose of the displaymay be calculated. The EKF may facilitate a retroactive update to apreviously rendered pose estimate stored in a rewind buffer. In someembodiments of the described technology the EKF may also oralternatively use a forward buffer to store current and predicted stateestimates. In some embodiments the method includes receiving a pluralityof measurement vectors from a rate-gyroscope, and selectively filteringthe measurement vectors based on a magnitude of each of the measurementvectors; a position state estimate may then be generated based upon thefiltered measurement vectors and other received data. In someembodiments the method applies an Extended Kalman Filter (EKF) to thereceived data and geodetic coordinates of a location of interest, toestimate the position of the location of interest relative to the poseof the display, based upon a retroactive adjustment of a previouslyrendered pose estimate using the delayed data measurements, arecalculation of the estimated position of the location of interestrelative to the pose of the display, and a prediction of future poseestimates by means of an adjust-update-predict (AUP) cycle.

The disclosed technology also regards a pose estimation system for usewith a display, including means for receiving sensor data from aplurality of sensors comprising delayed data measurements and datarepresenting the spatial position and orientation and orientation of adisplay, and means for applying an Extended Kalman Filter (EKF) to thereceived data and geodetic coordinates of a location of interest, toestimate a position of the location of interest relative to the pose ofthe display. The estimation may be based on a retroactive update to apreviously rendered pose estimate stored in a rewind buffer, usingdelayed sensor values. The estimation may also or alternatively use aforward buffer to store current and predicted state estimates. Thedisclosed technology may also include means for receiving a plurality ofmeasurement vectors from a rate-gyroscope, and selectively filtering themeasurement vectors based on a magnitude of each of the measurementvectors, wherein the means for generating a position state estimate isbased on the filtered measurement vectors and other received data. Insome embodiments of the disclosed technology the estimation of theposition of the location of interest may be based upon a retroactiveadjustment of a previously rendered pose estimate of the display usingthe delayed data measurements, a recalculation of the estimatedposition, and a prediction of future positions by means of anadjust-update-predict (AUP) cycle.

The disclosed technology also regards one or more tangiblecomputer-readable storage media encoding computer-executableinstructions for executing on a computer system a computer processincluding an extended Kalman Filter (EKF) prediction of a position andorientation of a display, using a forward buffer to store current andpredicted position estimates calculated by the computer process. Thepredicted position and orientation of the display may then be used bythe computer process to calculate the position of a location of interestrelative to the estimated position and orientation of the display.

In an application of the described technology, a user can quickly orienthimself and initiate movement as his next waypoint is automaticallyrendered in his environment as seen through the display (along withadditional pertinent information), rather than having to study his mapor check his compass. The user's azimuth, current position and objectivedestination may be constantly displayed. If asked to report hislocation, the user can give a status update without stopping orotherwise taking his attention away from his current field of view. Ifthe user encounters an obstacle to movement, he can easily detour orbypass the obstacle without losing his orientation to the movementobjective. His next waypoint, or other icon of interest, serves as adirectional beacon regardless of occlusions to his view or obstacles tohis forward progress.

Similarly, in a military application, soldiers are often required toeffectively manage airspace for numerous aircraft and to orchestrateaircraft movement in support of ground forces and sensor assets. Somemission profiles mandate unaided visual acquisition of an aircraft.Aircraft that broadcast their location can be geo-referenced by thedescribed technology, even if the aircraft is out of visual range. Oncein visual range, finding the aircraft can still be very challenging dueto occlusions (e.g., clouds, terrain) or other sensory challenges (e.g.,acoustic multi-path effects). The described technology enables rapidacquisition by the soldier of the locations of aircraft assets andprovides immediate understanding of aircraft attributes (e.g., callsign, altitude).

DRAWINGS

FIGS. 1A, 1B and 1C depict example embodiments of the system of thedescribed technology.

FIG. 2 is a peripheral view of an example embodiment of the system ofthe described technology.

FIG. 3 is a block diagram of the sensor package and the processingmodule of an example embodiment of the described technology.

FIG. 4 shows the various coordinate systems useful in the describedtechnology.

FIG. 5 is a qualitative timing diagram of the EKF processing of exampleembodiments of the described technology.

FIG. 6 shows a close-up look at an azimuth update based on arepresentative absolute orientation measurement useful in the system andmethods of the described technology; as depicted in the inset of thefigure, the EKF goes back in time using the rewind buffer to reprocessthe azimuth estimate based on the delayed absolute orientationmeasurement.

FIG. 7 shows an example embodiment of the user interface/display of thedescribed technology.

FIG. 8 depicts the predict-update (PU) cycle and the update-predict (UP)of an EKF method useful in the described technology.

FIG. 9 depicts an adjust-update-predict cycle of an exemplary embodimentof the described technology.

FIG. 10 is a block diagram representing an exemplary environment inwhich the present disclosure or parts thereof may be implemented.

FIG. 11 depicts the geometry behind the definition of the error measurein the augmented reality application of the described technology.

DETAILED DESCRIPTION

The augmented reality system of the described technology comprises insome embodiments a motion sensing and visualization kit 1, anaugmented-reality processing module 2 with a database 3, and may includea radio 4, as depicted in FIG. 1C and FIG. 2. The database may be remotefrom the visualization kit and the processing module

As shown in the embodiment of the described technology depicted in FIGS.1A, 1B and 1C and FIG. 2, the motion sensing and visualization kit 1 mayinclude a rigid reference assembly 11 with a camera (high-speed andhigh-resolution) 12 and a sensor package 13, and having a display 14with a graphical user interface 141 rendered on the display to conveyinformation in the form of text and graphics, an embodiment of which isshown in FIG. 7. In some embodiments of the system of the describedtechnology the display 14 is see-through. The sensors and processingmodule of the described technology can function with a wide variety ofdisplays, including by example and without limitation see-throughdisplays manufactured by the BAE, Lumus, and SA Photonics.

As depicted in FIG. 3, the sensor package 13 includes a plurality ofsensors, including a 3-axis accelerometer 131, a 3-axis angular-rategyro 132, a 3-axis magnetometer 133, a barometric pressure sensor 134, aGPS receiver 135 and a camera 12. The sensors may be mounted to therigid reference assembly as a packaged unit. While described as apackage, the sensors may be individually positioned about the rigidreference assembly 11 or on the user's clothing or other equipment, inaccordance with the technology as herein described. In some embodimentsthe rigid reference assembly is a helmet.

The sensors 13 are in wired communication (via a cable, or other hardwire) or wireless communication (via Bluetooth or other wirelesscommunication means) with the processing module 2 or other computationmeans of the described technology. As hereinafter described, theprocessing module processes data from the sensors and data from adatabase to generate display pose, and renders tactically-relevantinformation on the motion sensing and visualization kit's display 14. Insome embodiments the processing module is carried on the user's bodywhen the system is in operation. Coupled with the processing module is adatabase 3 including the geodetic coordinates of locations of interest(longitude, latitude and altitude), and digital terrain and elevationdata (DTED) to aid in the estimation of altitude. The processing modulemay further include custom software and standard libraries to receivegeo-spatial data (i.e., latitude, longitude and altitude informationabout objects in the environment) via a radio network or otherwise, andsoftware to render this data to a GUI 141.

The processing module or other computation means may be in wiredcommunication (via a cable, or other hard wire) or wirelesscommunication (via Bluetooth, or other wireless communications means)with the display 14. The processing module may also be coupled by wireor wireless communication with the radio 4, which receives signalsrelating to data in the database, and supports receiving and parsingXML-based messages from a digital radio network.

Further, as shown in FIG. 7, the user interface/display 141 may provideoperational alerts (e.g., notification that the radio network isinoperable, that the system is magnetically-disturbed, or the GPS signalis denied or degraded), system status information (e.g., user interfacemode ID, system battery level, operational time), system settings menuaccess, iconic visualization of geo-registered points of interest, and asituational awareness ring 213. A menu may not be displayed untilactivated by the user via a button/toggle switch located on or coupledwith the motion sensing and visualization kit 1; with the menu, the usermay access and change system configuration settings. The situationalawareness ring shown in FIG. 7 is an intuitive tool that offers the userin a military application a dynamic real-time 360° understanding ofwhere friendlies, enemies and other points of interest are located. Atthe center of the grid is the user's Military Grid Reference coordinate;located above the ring is the user's heading (on the fly configurable asmagnetic or true). Icons may move around the ring in response to userrotation. Geo-registered icons and those on the situational awarenessring are displayed in some embodiments with range information from theuser, and in some cases elevation (for aircraft icons). One or morebatteries may power various components of the system of the describedtechnology.

Suitable hardware for use in the processing module 2 include embeddedprocessing modules with, for example, an NVidia Tegra 3 system-on-chipand DDR3L memory. Similar suitable hardware may be found in currentcell-phone quad-core computing platforms.

Over time, in periodic intervals, the sensors of the sensor package 13measure various conditions, and transmit time-stamped signalsrepresenting the measurements to the processing module 2. Specifically,the accelerometer 131 provides a measure y_(a) of the difference betweenlinear acceleration of the sensor and the Earth's gravity vector, therate gyro 132 provides a measure u_(g) of angular rate, the magnetometer133 provides a measure y_(m) of the Earth's magnetic field to help indetermining azimuth, and the barometric pressure sensor 134 provides ameasure y_(bp) of barometric pressure for estimating altitude.Similarly, the GPS receiver 135 provides its position data y_(Gp)(latitude, longitude, and altitude) and its velocity data y_(Gv) (North,East, and Down velocities).

The processing module 2 or other computation means receives measurementsfrom the sensors 13, and calculates over time the position vector of alocation of interest s relative to the pose of the display 14.

The vector of the location of interest s calculated by the processingmodule is referred to as vector r_(ds) ^(d), representing the currentestimated position of s relative to the pose of the display, expressedin display coordinates. The calculations are based upon an ExtendedKalman Filter (EKF) structure, performed by an “EKF Calculator” storedin memory and executable by a processor to calculate state predictions.The EKF Calculator may include software and/or hardware elements, andmay be implemented in any tangible computer-readable storage media.“Tangible computer-readable storage media” includes, but is not limitedto, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM,digital versatile disks (DVD) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other tangible medium which can be used to storethe desired information and which can be accessed by mobile device orcomputer. In contrast to tangible computer-readable storage media,intangible computer-readable communication signals may embody computerreadable instructions, data structures, program modules or other dataresident in a modulated data signal, such as a carrier wave or othersignal transport mechanism. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal.

The Extended Kalman Filter (EKF) is a method for estimating a set ofquantities describing a system, referred to as system states, given aset of measurements and a pair of mathematical models. The measurementsinclude system data, typically collected by sensors as the systemoperates. The first mathematical model, referred to as the processmodel, describes the evolution of the system states over time. Thesecond mathematical model, referred to as the measurement model,describes how the measurement data is related to the states. Being onlyan approximation, each model must include a measure of its uncertainty,which the EKF uses to produce an estimate of the system states from theinformation at hand. It does so by starting from initial estimates ofthe system states and their uncertainty, valid at some initial time t₀,and using the process model to produce new estimates, called theprediction, valid at some later time t₁ when measurement data is alsoavailable. At time t₁, the new information provided by the measurementdata, weighted using the measurement model, is used to make an update tothe prediction, resulting in the final estimates of the system statesand their uncertainty at t₁. This prediction-update cycle continues aslong as measurement data is available, producing estimates of systemstates and associated uncertainty along the way.

In some embodiments of the described technology the EKF structureincludes a rewind buffer to integrate delayed measurements, or a forwardbuffer to store current and predicted state estimates calculated by theprocessing module, or both. Using the vectors r_(ds) ^(d) calculated bythe processing module, the processing module 2 or other computationmeans generates signals to render graphic symbology on the display 14,corresponding to the location of interest s, that appears to the user tobe attached to that real-world location (i.e., geo-registered). In someembodiments the system uses DOD-standard MIL-STD-2525C symbology, whichdistinguishes between hostile, friendly, neutral and unknown symbolsbased on shape and color.

Specifically, the processing module 2 or other computation means of thedescribed technology implements a baseline GPS/INS, and aids it withvision-based and other non-inertial absolute orientation informationwhen available. The baseline GPS/INS is designed to provide a nominallevel of performance when vision-aiding and other measurements are notavailable, and integrate them when available for improved performance,without assumptions about their availability or periodicity. The systemof the described technology may further utilize measurements andcomputer processes and analysis to address latency and enhancerobustness of the system in the presence of magnetic or dynamicdisturbances, as hereinafter described.

The EKF structure used in the processing module 2 of the describedtechnology further accommodates and aligns time-delayed measurementsusing a buffer scheme. In some embodiments the processing moduleselectively filters the rate-gyro measurement vector u_(g) transmittedto the processing module, based on the magnitude of the vector u_(g). Insome embodiments the processing module of the described technologyaugments the standard predict-update cycle of the EKF process with anadditional step to reduce prediction error and improve responsiveness torate gyro and accelerometer inputs, as hereinafter described.

By means of the processing module 2 or other computation means andassociated standard libraries accessible thereby, the GUI 141 renders asituational awareness ring 21 and one or more icons on the displayrepresenting positions of locations of interest, using the vector r_(ds)^(d).

Various coordinate systems are involved in the calculations of thedescribed technology, as depicted in FIG. 4. The body coordinate systemb is the reference for the motion sensing and visualization kit 1, withorigin at the point p. The camera coordinate system c consists of apermutation of the body coordinate system's axes and shares the sameorigin. The display coordinate system d and the accelerometer coordinatesystem a are both rigidly attached to the body coordinate system.Coordinate system a is the reference for the motion sensing and sensors.Coordinate system n is the North-East-Down (NED) reference fornavigation. The Earth-Centered Earth-Fixed (ECEF) coordinate system e isused to specify points in the environment. Coordinate system i is theEarth Centered Inertial (ECI) coordinate system, which is a goodapproximation of a true inertial reference in the context of thedescribed technology. The WGS-84 ellipsoid is used as the world model.

The processing module calculates the position vector r_(ds) ^(d) of apoint s in the environment with respect to the origin of d, expressed ind coordinates, using the formula:r _(ds) ^(d)=(C _(n) ^(e) C _(b) ^(n) C _(d) ^(b))^(T) [r _(es) ^(e)−(r_(ep) ^(e) +C _(n) ^(e) C _(b) ^(n) r _(pd) ^(b))]whereinr_(pd) ^(b) is the position of the origin of d with respect to p,expressed in b coordinates, obtained from physical measurements on theactual system or measurements on its solid-model drawing equivalent.r_(es) ^(e) is the position vector of a point s in the environment withrespect the origin of e, expressed in e coordinates, obtained byconverting the known latitude, longitude, and altitude of the point sfrom the database into equivalent Cartesian coordinates.r_(ep) ^(e) is the position of p with respect to the origin of e,expressed in e coordinates, obtained by converting the latitude,longitude, and altitude of the point p into equivalent Cartesiancoordinates. The latitude, longitude, and altitude of the point p areestimated by the system's EKF using sensor data, as hereinafterdescribed.

The position vectors of points s (p_(s)) and p (p_(p)) with respect tothe origin of e are specified in terms of latitude, L, longitude, λ, andaltitude, h. The position vector p_(s) is stored in the database 3; theposition vector p_(p) is calculated using the method as hereinafterdescribed. The conversion from latitude, longitude, and altitudecoordinates into their Cartesian equivalents is performed by theprocessing module of the described technology, by the mapping:x ^(e)=(R _(N)(L)+h)cos L cos λy ^(e)=(R _(N)(L)+h)cos L sin λz ^(e)=[1−e ²)(R _(N)(L)+h)] sin Lwherein R_(N) (L) and e are WGS-84 ellipsoid parameters.C_(n) ^(e) represents the orientation of the North-East-Down (n)coordinate system (see FIG. 4) with respect to the Earth-CenteredEarth-Fixed (e) coordinate system, obtained from a known coordinatetransformation that is a function of latitude, longitude, and altitudeof the point p. The latitude, longitude, and altitude of the point p areestimated by the system's EKF using sensor data.C_(b) ^(n) represents the orientation of the body coordinate system (b)(see FIG. 4) with respect to the North-East-Down (n) coordinate system.C_(b) ^(n) is estimated by the system's EKF using sensor data.C_(d) ^(b) represents the orientation of the display coordinate system(d) with respect to the body coordinate system (b) obtained froma-priori calibration based upon alignment of features in an imageacquired by the camera and expressed in body coordinates withcorresponding features in the same image expressed in displaycoordinates.

Once C_(b) ^(n) and p_(p) are estimated using the system's EKF, theprocessing module 2 renders the GUI information on the display 14 sothat an icon representing the position s can be rendered at displaycoordinates r_(ds) ^(d).

The EKF used in the processing module 2 of the described technology isbased upon the general model:

$\frac{d\; x}{d\; t} = {f\left( {x,u,w,t} \right)}$ŷ_(k) = h_(k)(x_(k), v_(k))where t is time, f is a continuous-time process, h_(k) is adiscrete-time measurement (with output ŷ_(k)), x is the state vector,x_(k) is its discrete-time equivalent, and u is the input vector. Thevector w is a continuous-time zero-mean white noise process withcovariance Q (denoted as w˜N(0,Q)) and v_(k) is a discrete-timezero-mean white-noise process with covariance R_(k) (denoted asv_(k)˜N(0,R_(k))).

The state is defined as x=[p_(p);v_(ep) ^(n);q_(nb);b_(g);b_(a)](semicolons are used to indicate column stacking), wherein v_(ep) ^(n)is the velocity of the point p with respect to the ECEF coordinatesystem, expressed in NED coordinates, and q_(nb) is the quaternionrepresentation of C_(b) ^(n). The vector b_(g) is the rate-gyro bias,and the vector b_(a) is the accelerometer bias. The rate gyro andaccelerometer data are inputs to the process model, so thatu=[u_(a);u_(g)], withu _(a) =f _(ip) ^(b) +b _(a) +w _(a)u _(g)=ω_(ib) ^(b) +b _(g) +w _(g)where f_(ip) ^(b)=(C_(b) ^(n))^(T)[a_(ep) ^(n)−g^(n)+(ω_(en)^(n)+2ω_(ie) ^(n))×v_(ep) ^(n)] is the specific force at p, ω_(ib) ^(b)is the angular rate of the body coordinate system with respect to theECI coordinate system (i), ω_(en) ^(n) is the angular rate of ncoordinate system with respect to the e coordinate system (expressed inn coordinates), ω_(ie) ^(n) is the angular rate of the e coordinatesystem with respect to the i coordinate system (also expressed in ncoordinates), w_(a)˜N(0,Q_(a)) and w_(g)˜N(0,Q_(g)). The cross productin the f_(ip) ^(b) expression is a Coriolis and centripetal accelerationterm due to motion over the Earth's surface, and can be neglected whenthe velocity is small (which is the case for pedestrian navigation).

Using the state definition and input model described above, the processmodel is specified by the following equations:{dot over (p)} _(p) =f _(p)(x)+w _(p){dot over (v)} _(ep) ^(n) =C _(b) ^(n)(u _(a) −b _(a) −w _(a))+g^(n)−(ω_(en) ^(n)+2ω_(ie) ^(n))×v _(ep) ^(n) +w _(v){dot over (q)} _(nb)=½Ω(q _(nb))(u _(g) −b _(g) −w _(g)−ω_(in) ^(b))+w_(q)b _(g) =w _(b) _(g)b _(a) =w _(b) _(a)where

${f_{p} = {\begin{bmatrix}\frac{1}{{R_{M}(L)} + h} & 0 & 0 \\0 & \frac{1}{\left( {{R_{N}(L)} + h} \right)\cos\; L} & 0 \\0 & 0 & {- 1}\end{bmatrix}v_{ep}^{n}}},$R_(M) and R_(N) are WGS-84 parameters, g^(n) is the acceleration due togravity, Ω is a 4×3 matrix that transforms an angular rate vector intothe corresponding quaternion derivative, and ω_(in) ^(b)=(C_(b)^(n))^(T) (ω_(ie) ^(n)+ω_(en) ^(n)). The process noise vector isw=[w_(p); w_(v); w_(q); w_(g); w_(b) _(g) ; w_(a); w_(b) _(a) ], and itscovariance matrix is Q=blkdiag (Q_(p), Q_(v), Q_(q), Q_(g), Q_(b) _(g) ,Q_(a), Q_(b) _(a) ). The measurement vector is defined as:

${\hat{y}}_{k} = {\begin{bmatrix}{\hat{y}}_{AO} \\{\hat{y}}_{a} \\{\hat{y}}_{m} \\{\hat{y}}_{Gv} \\{\hat{y}}_{Gp} \\{\hat{y}}_{D}\end{bmatrix} = \begin{bmatrix}{q_{nb} + v_{AO}} \\{{\left( C_{b}^{n} \right)^{T}\left( {a_{ep}^{n} - g^{n}} \right)} + b_{a} + v_{a}} \\{{\left( C_{b}^{n} \right)^{T}m^{n}} + v_{m}} \\{v_{ep}^{n} + v_{Gv}} \\{p_{p} + v_{Gp}} \\{h + v_{D}}\end{bmatrix}}$where ŷ_(AO) is an absolute orientation measurement, ŷ_(a) is theaccelerometer measurement, ŷ_(m) is the magnetometer measurement, ŷ_(Gv)is the velocity measurement, ŷ_(Gp) is the GPS horizontal position(i.e., latitude and longitude) measurement, and ŷ_(D) is the measurementof altitude based on DTED. The measurement noise vector isv_(k)=[v_(AO); v_(a); v_(m); v_(Gv); v_(Gp); V_(D)], and its covariancematrix is R_(k)=blkdiag (R_(AO), R_(a), R_(m), R_(Gv), R_(Gp), σ_(D) ²).

Because of the block-diagonal structure of R_(k), the EKF measurementupdate step is executed by processing measurements from each sensor asseparate sequential updates (in the same order as they appear in theŷ_(k) vector above).

The gravity vector is approximated as being perpendicular to the WGS-84ellipsoid and therefore modeled as g^(n)=[0; 0; g₀(L)], where the downcomponent g₀(L) is obtained from the known Somigliana model. Since theyare used as measurements of the gravity vector in body coordinates,accelerometer-based updates are only valid if the acceleration a_(ep)^(n) is zero. If not, these measurements are considered to be corruptedby an unknown dynamic disturbance. However, this disturbance isaddressed by detecting its presence and, consequently, increasing thecorresponding measurement noise covariance matrix, R_(a), by a largefactor ρ_(a) (e.g., ρ_(a)=100). Detection is based on comparing the normof the accelerometer measurement to ∥g^(n)∥, and also checking that themeasured angular rate is lower than a threshold whose value isapplication dependent (e.g., 3°/sec. in certain conditions). Inhead-worn applications, the location of the sensor package on the motionsensing and visualization kit, and the corresponding kinematics due tohead movement, result in angular rate being a very good indicator ofa_(ep) ^(n). The approach of increasing R_(a) implies that the unknownacceleration a_(ep) ^(n) is modeled as a stationary white noise process.Though the actual process is not stationary or white, it was foundexperimentally that this approach yields better results than thealternative of completely rejecting accelerometer measurements that aredeemed disturbed. In fact, when testing this alternative, it wasobserved that a single valid measurement after long periods of dynamicdisturbance (as in the case when walking) could cause undesirable jumpsin the estimates of b_(g) and b_(a), while increasing R_(a) resulted inno such issues.

Magnetometer-based measurement updates are valid if the magnetic fieldbeing measured is the Earth's magnetic field only. Otherwise, thesemeasurements are considered to be corrupted by an unknown magneticdisturbance. Therefore the processing module or other computation meansof the described technology may detect the presence of magneticdisturbances and, if detected, rejects the corresponding magnetometermeasurements. Detection is based on comparing the norm of the measuredmagnetic field vector y_(m) to the Earth's field strength B_(m), as wellas checking that the computed inclination angle is not too far (e.g.,0.5 deg) from the nominal value. Since it is based on the inner producty_(m) ^(T)y_(a), the latter check is only performed if no dynamicdisturbance is detected.

The processing module or other computation means may use a circularrewind buffer (RB) to maintain a record of relevant informationpertaining to the last N_(r), samples of EKF processing. This is done toproperly integrate absolute orientation measurements, which are delayedwith respect to the rest of the data (as depicted in FIG. 5, aqualitative timing diagram of the EKF processing of the describedtechnology). By this buffer, when absolute orientation information isprocessed and delivered, the EKF can reprocess past information. In theprocessing module 2 of the described technology the absolute orientationdata acquisition is synchronized with the sensor data acquisition. Thisreprocessing of past data is handled within a single EKF epoch Δt. FIG.6 shows a close-up look at an azimuth update based on a representativeabsolute orientation measurement. The EKF is able to “go back in time”and use the rewind buffer to reprocess the state estimate based on thelate measurement, all within its regular processing interval. In theexample illustrated in the inset in FIG. 6, the EKF goes back in timeusing the rewind buffer to reprocess the azimuth estimate of the pose ofthe display based on the delayed absolute orientation measurement.

The processing module may also or alternatively use a forward buffer(FB) to store both the current state estimate x_(k) ⁺ and the predictedstate estimates up to N_(f) time steps ahead. That is FB_(k)={x_(k)⁺,x_(k+1) ⁻,x_(k+2) ⁻, . . . ,x_(k+N) _(f) ⁻}. Through interpolation ofthe FB vectors, a state estimate can then be produced for anytε[t_(k),t_(k)+N_(f)Δt] where t_(k) is the time of the current estimateand Δt is the EKF's processing interval. Given a value Δt_(d) for systemlatency, the pose that is delivered at the time t_(k) for renderinggraphic on the display is based on the predicted state att=t_(k)+Δt_(d), which is extracted from the FB. N_(f) must be selectedsuch that N_(f)>0 and N_(f)Δt≧Δt_(d). Focused experiments have shown areduction in perceived latency from about 40 ms to about 2 ms when usingthis forward-prediction process.

Prior to use, the sensors 13 of the described technology must becalibrated. Hardware calibration of the motion sensing and visualizationkit consists of estimating C_(d) ^(b), r_(pd) ^(b), C_(a) ^(b), andr_(pa) ^(b). Estimation of the spatial position and relativeorientation, C_(a) ^(b), of the sensors with respect to the bodycoordinate system is performed, which also yields an estimate of thecamera's intrinsic parameters. Estimation of the relative orientation,C_(d) ^(b), of the display 14 with respect to the body coordinate systemis performed by an iterative process based on using an initial C_(d)^(b) estimate to render scene features (e.g., edges) from camera imageryonto the display 14, and adjusting it until the rendered features alignwith the corresponding actual scene features when reviewed through thedisplay 14. The position vectors r_(pd) ^(b) and r_(pa) ^(b) can beobtained by straightforward measurement, but in fact they are negligiblein the context of this application, the former because∥r_(pd)∥<<∥r_(ps)∥, and the latter because its magnitude is very smalland was empirically determined to have negligible effect. Themagnetometer 133 is also calibrated prior to each operation.

The initial state x(0) is estimated by using sensor readings during thefirst few seconds of operation before the EKF process starts. Theinitial condition of all biases is set to zero.

The processing module 2 uses a number of parameter values that have beentuned experimentally prior to system use. These are values for Q, R_(k),the initial estimated error covariance matrix P(0), and a number ofparameters that are used for disturbance detection, filtering, etc. Thistuning is performed by combining Allan variance analysis of sensor datawith the models herein described, to identify a starting point, and thenperforming a series of focused field experiments.

The forward-prediction process extrapolates motion to predict the stateat some time in the future, and is inherently sensitive to noise. Thismay result in jitter (i.e., high-frequency small-amplitude motion) ofthe rendered graphics even when the system is perfectly stationary(e.g., mounted on a tripod). Low-pass filtering of the rate gyro signal,u_(g), transmitted to the processing module reduces this jitter effectbut also introduces a time lag between real-world motion and motion ofthe corresponding symbology rendered on the display. Since this lag isnot noticeable when the rotation rate is near zero, and the jitter isnot noticeable when there is actual motion, in some embodiments thedescribed technology achieves a reduction in perceived jitter bylow-pass filtering the rate gyro signal only when the estimated rotationrate magnitude ∥u_(g)−b_(g)∥ is small (e.g., less than 5 deg/s). Asspecified below, this is done by adjusting the low-pass filter'sbandwidth using a smooth increasing function of estimated rotation ratemagnitude. The adaptive gyro filtering method is implemented in theprocessing module of the described technology by using the discrete-timefilter ũ_(g,k)=aũ_(g,k−1)+(1−a)u_(g,k) witha=Aexp(−0.5∥u_(g,k)−b_(g)∥²/σ_(a) ²) where 0≦A<1 and σ_(a)>0 areparameters that are established prior to use (e.g., A=0.85, σ_(a)=0.05).The resulting filtered signal can then be used in place of u_(g) in theEKF's time-propagation steps (i.e., in the forward-prediction process).In testing, your inventors found that this method reduces jitter by afactor of three without any adverse effects in other performancemeasures.

A single pose estimation processing step takes as inputs the currentsensor data, the RB data, and an index i_(now) corresponding to thecurrent-time location in the RB. It returns updates to RB, i_(now), andthe whole FB. An example of its implementation is as follows:

1: pre-process sensor data 2: RB[i_(now)] ← {sensor data, pre-processeddata} 3: i_(stop) = i_(now) 4: if vision data is available and ∃i_(vis): t_(CLK) in RB[i_(vis)] = t_(CLK) in vision data then 5:  i_(slow) =i_(vis) 6: end if 7: keep_processing = true 8: while keep_processing =true do 9:  {x⁻, P⁻} ← RB[i_(now)] 10:  RB[i_(now)] ← {x⁺, P⁺} =ekf_u(x⁻, P⁻, RB[i_(now)]) 11:  i_(next) = i_(now) + 1 12:  RB[i_(next)]← {x⁻, P⁻} = ekf_p(x⁺, P⁺, RB[i_(now)]) 13:  if i_(now) = i_(stop) then14:   FB[0] ← x⁺ , FB[1] ← x⁻ 15:   for k_(p) = 2 to N_(f) do 16:   {x⁻, P⁻} = ekf_p(x⁻, P⁻, RB[i_(now)]) 17:    FB[k_(p)] ← x⁻ 18:   endfor 19:   keep_processing = false 20:  end if 21:  i_(now) = i_(next)22: end whilewhere t_(CLK) is the reference time stamp of both sensor and vision dataacquisition, and lines 10 and 12 are the EKF measurement update andprediction steps, respectively. The loop on lines 15-18 implements theforward-prediction process by repeating single EKF prediction steps.

Accuracy performance is based on a measure of error, e, defined as theangle between the vectors r_(ps′) ^(b) and r_(pŝ) ^(b), as depicted inFIG. 11. The point s′ is the point in the undistorted camera imagecorresponding to the real-world reference point s, and is obtained viasemi-automatic processing (i.e., requiring some manual input) of theimagery. The vector r_(pŝ) ^(b) is the result of using the poseestimate, {p_(p), C_(b) ^(n)}, to compute r_(ps) ^(b). Note that, inaddition to pose estimation errors, the process of generating the‘ground-truth’ vector r_(ps′) ^(b), also contributes to ε. Thiscontribution was observed to be up to 3 mrad across a variety ofexperiments.

EKF implementations repeatedly perform either a predict-update (PU)cycle or an update-predict (UP) cycle (shown in FIG. 8). The differencebetween these cycles amounts to the time of availability of the stateestimate: the UP cycle implementation can produce an estimate soonerrelative to the time when sensor data is available. In either case onlycurrent measurement data y_(k) are utilized to estimate the currentstate (current input data u_(k) are not).

In some embodiments of the described technology, the processing moduleor other computation means adjusts the previous prediction by includingthe effects of the current input data, before executing the update step.This adjust-update-predict (AUP) cycle (shown in FIG. 9) has the effectthat both current measurement data y_(k) and current input data u_(k)are used to estimate the current state. Therefore, the AUP cycleimplementation is more responsive than the UP cycle to changing input u,provides a better estimate to the update step, and requires very littleadditional computation.

In some applications these systems and the methods herein describedcombine a novel pose estimation capability and a plurality of sensors toallow rendering of geo-registered graphics on a see-through display,thereby appearing to be part of the real environment as the user looksat the environment through a display.

The pose estimation systems and methods herein described can beimplemented in a wide variety of commercial and consumer applications.First-responder or search-and-rescue personnel can see geo-registeredicons representing the locations of team members, search regions, andkey objects of interest during mission operations. Accessing thisinformation in a heads-up posture enables the user to perform activitiesmore safely, with higher operational tempo, and with greater teamcoordination. Construction-site or warehouse foremen can view iconsrepresenting workers and material locations to help monitor safety onthe worksite and to support quick decision making about resourceallocation. Oil-and-gas industry workers can view graphics representinglocations of structures of interest, such as underground or underwaterpipelines, system safety components, and graphical representation ofimportant system states (e.g., pressure and temperature of storage tanksand pipeline sections). Outdoor recreational enthusiasts (e.g., runners,bicyclists, hikers) can be presented with directional information,waypoints, and details about their exact position and heading whencarrying out active movement while in a heads-up posture viewing thereal-world environment. For immersive training applications, users canbe presented with virtual avatars that appear as part of theirreal-world environment as they maneuver and carry out training scenariosindoors or outdoors. Such training enables the user to practice andimprove scenario-specific decision making. This immersive training maybe extended to sports training applications, where athletes may useaugmented and/or virtual reality to enhance their training program. Thepose estimation systems and methods herein may also be applied to gamingscenarios where augmented reality and/or virtual reality is used toenhance user experience and the estimation of pose of a gaming device isrequired. Other applications include the transportation industry, wherevehicle operators may access information that appears to part of thereal-world environment, and maintenance personnel may view pertinentinformation overlaid on the system under maintenance/repair.

Further, the pose estimation system and methods as herein described canbe implemented with a variety of display technologies, including nightvision goggles, see-through displays, wearable smart glasses, andsmartphone or tablet devices. For smartphone or tablet styleapplications, the position and orientation of the smartphone or tabletis accurately tracked while the user holds the device in an uprightposition in their line of sight to view the real-world while ‘lookingthrough the phone’. In this video see-through application,geo-registered graphical information is superimposed on the device'scamera imagery and presented to the user real-time on the device'sdisplay.

The invention claimed is:
 1. A pose estimation system for use with adisplay, the system comprising: a. a plurality of sensors, b. a databasecomprising geodetic coordinates of a location of interest, and c. aprocessing module communicatively coupled with the database and thesensors, the processing module receiving over time data from thesensors, and data representing the spatial position of the displayrelative to the spatial position of the sensors, and applies an ExtendedKalman Filter (EKF) to the received data to estimate the position andorientation of the display, using a forward buffer to store current andpredicted position estimates calculated by the processing module, andcalculates a position of the location of interest relative to theposition of the display, wherein the position of the location ofinterest is the vector r_(ds) ^(d), calculated using the formula:r _(ds) ^(d)=(C _(n) ^(e) C _(b) ^(n) C _(d) ^(b))^(T) [r _(es) ^(e)−(r_(ep) ^(e) +C _(n) ^(e) C _(b) ^(n) r _(pd) ^(b))] wherein r_(pd) ^(b)is the position of the origin of the display coordinate system d withrespect to the point of origin of the body coordinate system p,expressed in body system coordinates b; r_(es) ^(e) is the positionvector of the location of interest s in the environment with respect tothe origin of the earth centered, earth fixed coordinate system e,expressed in e coordinates, derived from the geodetic coordinates of thelocation of interest in the database; and r_(ep) ^(e) is the position oflocation p with respect to the origin of e, expressed in e coordinates,wherein C_(n) ^(e) is an orientation of a North-East-Down (n) coordinatesystem with respect to e, wherein C_(b) ^(n) is an orientation of b withrespect to the North-East-Down (n) coordinate system, and wherein C_(d)^(b) is the orientation of the display relative to the body systemcoordinates b.
 2. The pose estimation system of claim 1, wherein thedatabase is remote from the sensors and the processing module.
 3. Thepose estimation system of claim 1, wherein the plurality of sensorscomprise a camera, a 3-axis accelerometer, a 3-axis angular rate gyro, a3-axis magnetometer, a barometric pressure sensor, and a GPS receiver.4. The pose estimation system of claim 1, wherein the database furthercomprises digital terrain and elevation data, and the processing moduleingests said data to aid in the estimation of altitude of the display.5. The pose estimation system of claim 1, wherein the plurality ofsensors are mounted to a rigid reference assembly.
 6. The poseestimation system of claim 1, further comprising a display, and whereingraphical user interface information comprising a graphical iconrepresenting the location of interest is rendered on the display at theposition estimated by the processing module.
 7. The pose estimationsystem of claim 1, further comprising a radio configured to send andreceive geo-spatial data relating to objects in the environment, whereinthe processing module receives, stores and uses the geo-spatial data torender graphical user interface information on the display comprising agraphical icon representing some of the objects in the environment. 8.The pose estimation system of claim 1, wherein the processing moduleimplements a baseline GPS/INS to estimate the position of the locationof interest, and adjusts the GPS/INS values over time with absoluteorientation information.
 9. The pose estimation system of claim 1,wherein the plurality of sensors comprises a 3-axis magnetometer, andwherein the processing module compares the norm of data from the 3-axismagnetometer to the Earth's field strength to detect the presence orabsence of magnetic disturbances and, if detected, rejects data from themagnetometer corresponding to the times of detected magneticdisturbances.
 10. The pose estimation system of claim 1, wherein theplurality of sensors comprises a 3-axis magnetometer, and wherein theestimation means is designed and configured to compare the norm of datafrom the 3-axis magnetometer to the Earth's field strength to detect thepresence or absence of magnetic disturbances and, if detected, rejectdata from the magnetometer corresponding to the times of detectedmagnetic disturbances.
 11. A method for providing an estimate ofposition for a location of interest relative to a display, the methodcomprising the steps of: a. receiving sensor data from a plurality ofsensors comprising delayed data measurements and data representing thespatial position and orientation of a display, b. applying an ExtendedKalman Filter (EKF) to the received data and geodetic coordinates of alocation of interest, to estimate a position of the location of interestrelative to the position and orientation of the display, using a forwardbuffer to store current and predicted position estimates of the display,wherein the position of the location of interest is the vector r_(ds)^(d), calculated using the formula:r _(ds) ^(d)=(C _(n) ^(e) C _(b) ^(n) C _(d) ^(b))^(T) [r _(es) ^(e)−(r_(ep) ^(e) +C _(n) ^(e) C _(b) ^(n) r _(pd) ^(b))] wherein r_(pd) ^(b)is the position of the origin of the display coordinate system d withrespect to the point of origin of the body coordinate system p,expressed in body system coordinates b; r_(es) ^(e) is the positionvector of the location of interest s in the environment with respect tothe origin of the earth centered, earth fixed coordinate system e,expressed in e coordinates, derived from the geodetic coordinates of thelocation of interest in a database; and r_(ep) ^(e) is the position oflocation p with respect to the origin of e, expressed in e coordinates,wherein C_(n) ^(e) is an orientation of a North-East-Down (n) coordinatesystem with respect to e, wherein C_(b) ^(n) is an orientation of b withrespect to the North-East-Down (n) coordinate system, and wherein C_(d)^(b) is the orientation of the display relative to the body systemcoordinates b.
 12. The method of claim 11, wherein the sensor datareceived comprises time-stamped sensor values comprising camera imagery,accelerometer measurements, rate-gyro measurements, magnetometermeasurements, barometric pressure sensor measurements, GPS receiverposition measurements, and GPS receiver velocity measurements.
 13. Apose estimation system for use with a display, the system comprising: a.means for receiving sensor data from a plurality of sensors comprisingdelayed data measurements and data representing the spatial position ofa display, b. means for applying an Extended Kalman Filter (EKF) to thereceived data and geodetic coordinates of a location of interest, toestimate a position and orientation of the display, using a forwardbuffer to store current and predicted position estimates, and calculatethe position of a location of interest relative to the position andorientation of the display, wherein the position of the location ofinterest is the vector r_(ds) ^(d), calculated using the formula:r _(ds) ^(d)=(C _(n) ^(e) C _(b) ^(n) C _(d) ^(b))^(T) [r _(es) ^(e)−(r_(ep) ^(e) +C _(n) ^(e) C _(b) ^(n) r _(pd) ^(b))] wherein r_(pd) ^(b)is the position of the origin of the display coordinate system d withrespect to the point of origin of the body coordinate system p,expressed in body system coordinates b; r_(es) ^(e) is the positionvector of the location of interest s in the environment with respect tothe origin of the earth centered, earth fixed coordinate system e,expressed in e coordinates, derived from the geodetic coordinates of thelocation of interest in a database; and r_(ep) ^(e) is the position oflocation p with respect to the origin of e, expressed in e coordinates,wherein C_(n) ^(e) is an orientation of a North-East-Down (n) coordinatesystem with respect to e, wherein C_(b) ^(n) is an orientation of b withrespect to the North-East-Down (n) coordinate system, and wherein C_(d)^(b) is the orientation of the display relative to the body systemcoordinates b.
 14. The pose estimation system of claim 13, wherein theplurality of sensors comprise a camera, a 3-axis accelerometer, a 3-axisangular rate gyro, a 3-axis magnetometer, a barometric pressure sensor,and a GPS receiver.
 15. The pose estimation system of claim 13, furthercomprising means to display a graphic icon at a location of interestrelative to the position and orientation of the display means, asestimated by the estimation means.
 16. The pose estimation system ofclaim 13, further comprising means for sending and receiving geo-spatialdata relating to objects in the environment, wherein the estimationmeans is further configured to receive, store and use the geo-spatialdata to generate signals for rendering graphical user interfaceinformation on a display.
 17. The pose estimation system of claim 13,wherein the estimation means further implements a baseline GPS/INS toestimate the position of a location of interest relative to a display,and adjusts the GPS/INS values over time with absolute orientationinformation.